Modeling and prediction of driver behavior by foot gesture analysis

Please cite this article in press as: C. Tran et a (2011), doi:10.1016/j.cviu.2011.09.008 Understanding driver behavior is an essential component in human-centric Intelligent Driver Assistance Systems. Specifically, driver foot behavior is an important factor in controlling the vehicle, though there have been very few research studies on analyzing foot behavior. While embedded pedal sensors may reveal some information about driver foot behavior, using vision-based foot behavior analysis has additional advantages. The foot movement before and after a pedal press can provide valuable information for better semantic understanding of driver behaviors, states, and styles. They can also be used to gain a time advantage in predicting a pedal press before it actually happens, which is very important for providing proper assistance to driver in time critical (e.g. safety related) situations. In this paper, we propose and develop a new vision based framework for driver foot behavior analysis using optical flow based foot tracking and a Hidden Markov Model (HMM) based technique to characterize the temporal foot behavior. In our experiment with a real-world driving testbed, we also use our trained HMM foot behavior model for prediction of brake and acceleration pedal presses. The experimental results over different subjects provided high accuracy ( 94% on average) for both foot behavior state inference and pedal press prediction. By 133 ms before the actual press, 74% of the pedal presses were predicted correctly. This shows the promise of applying this approach for real-world driver assistance systems. 2011 Elsevier Inc. All rights reserved.